Automated metamorphic testing of variability analysis tools

摘要

Variability determines the capability of software applications to be configured and customized. A commonneed during the development of variability–intensive systems is the automated analysis of their underlyingvariability models, e.g. detecting contradictory configuration options. The analysis operations that areperformed on variability models are often very complex, which hinders the testing of the correspondinganalysis tools and makes difficult, often infeasible, to determine the correctness of their outputs, i.e.the well–known oracle problem in software testing. In this article, we present a generic approach forthe automated detection of faults in variability analysis tools overcoming the oracle problem. Our workenables the generation of random variability models together with the exact set of valid configurationsrepresented by these models. These test data are generated from scratch using step–wise transformationsand assuring that certain constraints (a.k.a. metamorphic relations) hold at each step. To show the feasibilityand generalizability of our approach, it has been used to automatically test several analysis tools in threevariability domains: feature models, CUDF documents and Boolean formulas. Among other results, wedetected 19 real bugs in 7 out of the 15 tools under test.